Systems and methods for placing a gate and/or a color box during ultrasound imaging

ABSTRACT

A method for positioning one or both of a gate and a color box on an ultrasound image generated during scanning of an anatomical feature using an ultrasound scanner comprises deploying an artificial intelligence (AI) model to execute on a computing device communicably connected to the ultrasound scanner, wherein the AI model is trained so that when the AI model is deployed, the computing device generates a prediction of at least one of an optimal position, size, or angle for the gate and/or an optimal location/size of the color box on the ultrasound image generated during ultrasound scanning of the anatomical feature, thereafter enabling the acquisition of corresponding Doppler mode signals.

FIELD

The present disclosure relates generally to ultrasound imaging, and inparticular, systems and methods for placing a gate or a color box duringultrasound imaging.

BACKGROUND

Ultrasound is a useful, non-invasive imaging technique capable ofproducing real time images. Ultrasound imaging has an advantage overX-ray imaging in that ultrasound imaging does not involve ionizingradiation.

Ultrasound imaging systems may generally be operated in various Dopplermodes that take advantage of the fact that reflected echoes undergo achange in frequency when reflected by moving objects in tissue (e.g.,blood in vascular tissue). Some Doppler modes include: spectral Doppler,pulsed wave (PW) Doppler, continuous wave (CW) Doppler, color Doppler,and Power Doppler. Tissue Doppler Imaging (TDI) is also a particular wayof using spectral or Color Doppler for visualizing tissue wall motionusing a lower frequency signal acquisition rate. It can be interchangedwith the use of PW Doppler and Color Doppler as necessary.

When an ultrasound scanner is used in a PW Doppler mode, it allows theoperator to select a specific, small area on the image, and, in thetissue corresponding to that area, measure blood motion velocity. Aspart of this process, a gate is specified by the user, along anultrasound beam line or direction (e.g., a one-dimensional signal isobtained). At the gate location, an algorithm is applied to processhigh-pass filtered, demodulated data into a Fourier transform, in orderto look at low-frequency motion of structures, such as blood, within thegate. The result is a spectrum as a function of time that shows thegeneral velocity at the gate location. Color doppler providesinformation about the presence or absence of flow, mean flow velocityand direction of flow within a selected color box on an anatomicalfeature. Spectral Doppler differs from Color Doppler imaging in thatinformation is not obtained from the entire color box (as placed) butfrom a specified gate window, as noted above, a generally 2-4 mm widesample volume.

Traditionally, ultrasound exams on vascular anatomy may include thesteps of imaging a vessel in brightness mode (B-mode), then placing aColor box, then positioning a gate where an operator desires to measureDoppler velocity. These various steps are typically performed manuallyby the operator, in a way that is inefficient for the ultrasoundoperator.

One of the key drawbacks and limitation of Doppler is inconsistentplacement of both gate and color box, where blood velocity is to bemeasured. Manual placement may be not only inefficient, as noted above,but vastly inconsistent between sonographers or even for the samesonographer, at different times. This variation may result in gatheringless diagnostically useful information. In fact, even a slight offset ingate angle (also referred to as the “correction angle”) can lead to upto 30% difference in accuracy of results. Generally, to evaluate anartery, the best angle for evaluation would be at zero degrees (parallelto the vessel) i.e. strongest signal and best waveforms would be at zerodegrees. Zero degrees is not usually clinically feasible, however, soinstead the probe is oriented at some angle between 0 (parallel) and 90degrees (perpendicular) when evaluating the vessel (usually between 30and 60 degrees).

By way of further background, to appreciate the criticality of accurategate placement, it is to be understood that ultrasound systems calculatethe velocity of blood flow according to the Doppler equation (theFourier Transform):

${{{\Delta\; f} = \frac{2\text{?}V\text{?}\theta}{C}},{\text{?}\text{indicates text missing or illegible when filed}}}\mspace{346mu}$

where Δf is the Doppler shift frequency, f₀ is the transmittedultrasound frequency, Vis the velocity of reflectors (red blood cells),θ (theta, the Doppler gate angle) is the angle between the transmittedbeam and the direction of blood flow within the blood vessel (thereflector path), and C is the speed of sound in the tissue (1540 m/sec).Since the transmitted ultrasound frequency and the speed of sound in thetissue are assumed to be constant during the Doppler sampling, theDoppler shift frequency is directly proportional to the velocity of redblood cells and the cosine of the Doppler angle. The angle θ affects thedetected Doppler frequencies. At a Doppler angle of 0°, the maximumDoppler shift will be achieved since the cosine of 0° is 1. Conversely,no Doppler shift (no flow) will be recorded if the Doppler angle is 90°since the cosine of 90° is 0.

The orientation of anatomical features and tissues through which bloodflows (for example, carotid arteries) may vary from one patient toanother; therefore, the operator is required to align the Doppler angleparallel to the vector of blood flow by applying the angle correction orangling the transducer. If the Doppler angle is small)(<50°, thisuncertainty leads to only a small error in the estimated velocity. IfDoppler angles of 50° or greater are required, then precise adjustmentof the angle correct cursor is crucial to avoid large errors in theestimated velocities. The Doppler angle should not exceed 60°, asmeasurements are likely to be inaccurate. For carotid arteries, apreferred angle of incidence is 45°±4. By way of example, in specificregard to carotid arteries, consistent use of a matching Doppler angleof incidence for velocity measurements in the common carotid artery andthe internal carotid artery reduces errors in velocity measurementsattributable to variation in θ. It is known in the art that operatorerrors and inconsistencies have made this area of ultrasound technologya challenge.

Furthermore, the optimal position of a color box in a normal artery isin the mid lumen parallel to the vessel wall, whereas in a diseasedvessel it should ideally be aligned parallel to the direction of bloodflow. In the absence of plaque disease, the color box should generallynot be placed on the sharp curves of a tortuous artery, as this mayresult in a falsely high velocity reading. If the color box is locatedtoo close to the vessel wall, artificial spectral broadening isinevitable. Leaving the specific positioning of the color box entirelyto operator judgment can lead to unnecessary errors.

There it can be appreciated that is thus a need for improved ultrasoundsystems and methods for placing a gate and/or a color box duringultrasound imaging of any anatomical feature and tissue through whichblood flows. The above background information is provided to revealinformation believed by the applicant to be of possible relevance to thepresent invention. No admission is necessarily intended, nor should beconstrued, that any of the preceding information constitutes prior artagainst the present invention. The embodiments discussed herein mayaddress and/or ameliorate one or more of the aforementioned drawbacksidentified above. The foregoing examples of the related art andlimitations related thereto are intended to be illustrative and notexclusive. Other limitations of the related art will become apparent tothose of skill in the art upon a reading of the specification and astudy of the drawings herein.

BRIEF DESCRIPTION OF THE DRAWINGS

Non-limiting examples of various embodiments of the present disclosurewill next be described in relation to the drawings, in which:

FIG. 1 is a user interface showing a B-mode image and a PW Doppler modespectrum, according to an embodiment of the present invention;

FIG. 2 is a diagram of a method for training and deployment of an AImodel for placement of a gate during ultrasound imaging, according to anembodiment of the present invention;

FIG. 3 is a diagram of a method for training and deployment of an AImodel for placement of a color box during ultrasound imaging, accordingto an embodiment of the present invention;

FIG. 4 is a schematic diagram of an ultrasound imaging system accordingto an embodiment of the present invention;

FIG. 5 is a schematic diagram of a distributed network of ultrasoundimaging systems according to an embodiment of the present invention;

FIG. 6 is a diagram of a method for training and deployment of an AImodel for both the placement of a gate and a color box during ultrasoundimaging, according to an embodiment of the present invention;

FIG. 7 is flowchart diagram of the steps for training a gate placementAI model, according to an embodiment of the present invention;

FIG. 8 is flowchart diagram of the steps for training a color boxplacement AI model, according to an embodiment of the present invention;

FIG. 9 is a flowchart diagram of an example method of image acquisition,processing employing an AI model, and placement of optimized gate(optimized gate parameters) on an anatomical feature shown in ultrasoundimage;

FIG. 10 is a flowchart diagram of an example method of imageacquisition, processing employing an AI model, placement of optimizedgate on an anatomical feature shown in ultrasound image, acquiringDoppler signal and subsequent interruption of Doppler signal to acquireupdated optimized gate parameters, using AI model;

FIG. 11 is a is a schematic diagram of a vascular image (Dopplerultrasound) showing labeled boundaries (two masks) for identifying,labeling and training a gate on an AI model, according to an embodimentof the present invention.

FIGS. 12-14 show a sequence of displays on the user interface as thesystem is switched between the PW Doppler mode and B-mode, for AI modelupdating of a gate on the anatomical feature, according to an embodimentof the present invention; and

FIG. 15 is a touchscreen user interface showing a B-mode image, with AImodel directed gate and color box placement, along with drop-down screenmode options, according to an embodiment of the present invention.

Unless otherwise specifically noted, articles depicted in the drawingsare not necessarily drawn to scale.

DETAILED DESCRIPTION A. Exemplary Embodiments

The system of the present invention uses a transducer (a piezoelectricor capacitive device operable to convert between acoustic and electricalenergy) to scan a planar region or a volume of an anatomical feature.Electrical and/or mechanical steering allows transmission and receptionalong different scan lines wherein any scan pattern may be used.Ultrasound data representing a plane or volume is provided in responseto the scanning. The ultrasound data is beamformed, detected, and/orscan converted. The ultrasound data may be in any format, such as polarcoordinate, Cartesian coordinate, a three-dimensional grid,two-dimensional planes in Cartesian coordinate with polar coordinatespacing between planes, or other format. The ultrasound data is datawhich represents an anatomical feature sought to be assessed andreviewed by a sonographer.

In one embodiment there is provided a method and system for a trained AImodel to position a gate, and wherein the system includes a spectralDoppler detector. In another embodiment there is provided a method andsystem for a trained AI model to position a color gate, and wherein thesystem includes a spectral Doppler detector. In another embodiment thereis provided a method and system for a trained AI model to position botha gate and color box, and wherein the system includes a spectral Dopplerdetector.

At a high level, the present embodiments are generally directed to anautomated way to position one or more of: a color box, a gate (e.g., agate for PW Doppler imaging), correction angle and gate size on anultrasound image. The embodiments automate the act of positioning thecolor box and PW gate parameters to remove the number of steps requiredto perform ultrasound examination of anatomical features and tissuesthrough which blood flows. Since these various steps are typicallyperformed manually by the operator, the present embodiments use thesemanual and other inputs to train an artificial intelligence (AI) modelto learn the area on ultrasound images where these user interface itemsare placed, so as to predict the location automatically on subsequentnew ultrasound image acquisitions.

The embodiments herein generally allow for the provision of ultrasoundsystems, ultrasound-based methods, computer-readable media storingcomputer-readable instructions, and portable computing devices forpositioning a color box and/or a gate (including gate location, size andangle) on an ultrasound image of a feature of interest, for examplearteries, for detecting medical conditions and anomalies therein.

Cerebrovascular disease (stroke) is the third leading cause of death inthe United States, accounting for over 400,000 new cases diagnosed eachyear. Ultrasonography of the carotid arteries is the modality of choicefor triage, diagnosis, and monitoring of cases of atheromatous disease.Important factors in diagnosis of atherosclerotic disease of theextracranial carotid arteries are the intima-media thickness, plaquemorphology, criteria for grading stenosis, limiting factors such as thepresence of dissection or cardiac abnormalities, distinction betweennear occlusion and total occlusion, and the presence of a subclaviansteal. Challenges to the consistency of carotid ultrasound results mayinclude poor Doppler technique including, as noted above, improper andinconsistent placement of the (Doppler) gate and/or the color box, evenby experienced sonographers. These issues may be overcome within thescope of the present invention, by largely removing i) gate placementparameters and/or ii) color box location, orientation and size from auser/sonographer's control and instead employing one or more AI modelstrained to do so.

In one aspect, the present invention provides a method for positioning agate on an ultrasound image generated during scanning of an anatomicalfeature using an ultrasound scanner, said gate at least defining anoptimal location of a Doppler mode signal in a tissue, the methodcomprising deploying an artificial intelligence (AI) model to execute ona computing device communicably connected to the ultrasound scanner,wherein the AI model is trained so that when the AI model is deployed,the computing device generates a prediction of at least one of anoptimal position, size, or angle for the gate on the ultrasound imagegenerated during ultrasound scanning of the anatomical feature;acquiring, at the computing device, a new ultrasound image duringultrasound scanning; processing, using the AI model, the new ultrasoundimage to generate a prediction of one or more of an optimal gateposition, size and angle (the “predicted optimized gate”); and employingthe predicted optimized gate to enable corresponding Doppler modesignals.

In another aspect, the present invention provides A method forpositioning a color box on an ultrasound image generated duringultrasound scanning of an anatomical feature, said color box at leastdefining an optimal location of a color Doppler mode signal in a tissue,the method comprising deploying an artificial intelligence (AI) model toexecute on a computing device communicably connected to the ultrasoundscanner, wherein the AI model is trained so that when the AI model isdeployed, the computing device generates a prediction of optimal colorbox placement for the color box, on the ultrasound image, duringultrasound scanning of the anatomical feature; acquiring, at thecomputing device, a new ultrasound image during ultrasound scanning;processing, using the AI model, the new ultrasound image to generate aprediction of the optimal new color box position; and employing the newcolor box position to enable corresponding color Doppler mode signals.

In another aspect, the present invention provides the creation anddeployment of an AI model which is trained to optimally place both agate (position, size and angle) and color box on the ultrasound imagegenerated during ultrasound scanning of an anatomical feature.

In still further aspects of the present invention, there are providedmethods of training AI models, as described herein, to optimize theiraccuracy, in the placement of one or both of a gate (position, size andangle) and color box on the ultrasound image generated during ultrasoundscanning of an anatomical feature.

In still further aspects of the present invention, there are provided amethod of updating the gate (the previously set gate) as follows:displaying on a user interface of the computing device a live spectralDoppler mode (“SD-mode”) ultrasound spectrum that corresponds to thepredicted optimized gate; receiving input to update to a new predictedoptimized gate; capturing a two-dimensional (2D) imaging mode (“2Dmode”) ultrasound image (“captured image”); applying the AI model to thecaptured image to generate a prediction of one or more of an optimalupdated gate position, size and angle (the “updated optimized gate”);employing the updated optimized gate to enable corresponding SD-modesignals; and displaying a live-SD mode ultrasound spectrum thatcorresponds to the updated optimized gate.

It is to be understood that “feature” (used interchangeably herein with“anatomical feature”) as used herein and to which the gate and color boxplacement embodiments of the invention may be applied, (for example, themethods, processes and systems described herein), is, broadly andwithout limitation, any anatomical feature and tissue through whichblood flows and in which measurement of blood flow is desired. As such,“feature” comprises the vascular system and the cardiovascular system.With the vascular system, arteries include, but are not limited to thegroup consisting of carotid artery, subclavian artery, axillary artery,brachial artery, radial artery, ulnar artery, aorta, hypergastic artery,external iliac artery, femoral artery, popliteal artery, anterior tibialartery, arteria dorsalis celiac artery, cystic artery, common hepaticartery (hepatic artery proper, gastric duodenal artery, right gastricartery), right gastroepiploic artery, superior pancreaticoduodenalartery, inferior pancreaticoduodenal artery, pedis artery, posteriortibial artery, ophthalmic artery and retinal artery. Within thecardiovascular system, “feature” includes but is not limited to theheart (including fetal heart) and gate placement in or around heartvalves. The term “feature” additionally comprises an umbilical cord.

Referring to FIG. 1, an example display device 6 (e.g., a tabletcomputer, smartphone, or the like, which may be communicably coupled toan ultrasound scanner), with screen 8 is shown on which a B-mode image 2is displayed. The B-mode image 2 includes features of a body, such asblood vessel walls 12, 14 and skin 16. Also displayed on the B-modeimage 2 is a gate 17 that indicates where a Doppler mode signal in thetissue corresponding to the gate location is obtained. The extent of thegate 17 is defined by ends 18, 19, and the direction of the gate 17 isdefined by line 20.

Typically, the blood vessel under observation is not in line with theultrasound line, and so additional lines next to the gate are shown toindicate a correction angle for the PW Doppler signal. The additionallines should generally be positioned parallel to the vessel walls. Theideal Doppler signal is parallel with the blood flow, and, at the otherextreme, a Doppler signal is unobtainable if the blood flow is entirelyperpendicular to the ultrasound line. The position and angle of the gatecan be adjusted to best orient for the particular ultrasound image, andthe correction angle (also referred to as gate angle herein) can be setto provide additional information to the system about the angle of thevessel side walls, so that the Doppler signal can be correctedaccordingly. In FIG. 1, the correction lines 21, 22 are shown positionedparallel to the walls 12, 14 of the blood vessel being scanned.

Also displayed on the touchscreen 8 is a Doppler mode display portion23, which shows a corresponding Doppler mode spectrum 24 that representsvelocity of blood flow on vertical axis 26 versus time on horizontalaxis 28. The displayed spectrum 24 moves to the left of the Doppler modedisplay portion 23 as time progresses, in the direction of block arrow34. The user interface of FIG. 1 is shown as an example only, and otherconfigurations of user interface items may be possible in differentembodiments.

Traditionally, the placement of the gate 17 on the B-mode image 2 isperformed manually, along with manual inputs to resize the ends 18, 19of the gate, as well the correction lines 21, 22 specifying thecorrection angle. Modifying these various user interface items to obtainthe desired Doppler signal to be displayed (e.g., in the display portion23) may take time. In certain instances, the operator may also need tore-adjust these items to optimize the Doppler signal to be displayed.

Additionally shown on the user interface of FIG. 1 is Update Gate button35, which enables a user to provide input to direct the AI model togenerate a prediction of one or more of an optimal updated gateposition, size and angle (the “updated optimized gate”). Such anupdating feature is described in further detail below and specificallyin FIGS. 12-14.

Referring to FIG. 2, shown there is a diagram of a method for trainingand deployment of an AI model for placement of a gate during ultrasoundimaging, according to an embodiment of the present invention. Generally,a software module may be provided on the display device 6 that runs inparallel to standard ultrasound imaging software used to receive inputsfrom the operator and display live ultrasound images to the user. Thissoftware module may track where a gate is manually placed by an operatorwhen in PW Doppler mode. Along with the gate position, it may also trackthe gate size, and/or correction angle when it is positioned on theB-mode image.

As shown in FIG. 2, these various inputs are shown generally asultrasound images 202 and 203 which includes the underlying B-modeimage, along with a manually-inputted gate position, gate size, and thecorrection angle. It is to be understood that the inputs for trainingstep 204 need not be B-mode images and may in fact be Doppler images aswell. These various user inputs may be collected as a “snapshot” of thesettings used by the ultrasound imaging system for generating theDoppler imagine desired by the operator. In some embodiments, thesevarious inputs may be collected as training data each time the userchanges the gate attributes. In some embodiments, a timer can be set sothat the snapshot of these various inputs is only treated as trainingdata if they are not immediately changed again within some set timeperiod (e.g., 0.25-2 seconds). In this latter scenario, the lack ofchange in the inputs after the predetermined period of time likely meansthat the user has finalized the input for visualizing the Doppler data,so that these inputs are more likely to be accurate and therefore usefulas training data.

The training ultrasound frames (202-203), which may be B-mode or Dopplerimages may include ultrasound frames labeled as Acceptable with gateparameters that are tagged as acceptable and representative of anoptimal gate location and/or size and/or angle and ultrasound frameslabeled as Unacceptable that are tagged respectively as unacceptable andunrepresentative of an optimal gate location and/or size and/or angle.Both the training ultrasound frames labeled as Acceptable andUnacceptable may themselves be used for training and/or reinforcing AImodel 206. This is shown in FIG. 2 with tracking lines from ultrasoundimages labeled as both Acceptable and Unacceptable, to trainingalgorithm step 204.

In some embodiments, an optional pre-processing act 201 may be performedon the underlying ultrasound image frames 202 and 203 to facilitateimproved performance and/or accuracy when training the machine learning(ML) algorithm. For example, it may be possible to pre-process theultrasound images 202 and 203 through a high contrast filter to reducethe granularity of greyscale on the ultrasound images 202 and 203.

Additionally, or alternatively, it may be possible to reduce scale ofthe ultrasound images 202-203 prior to providing the ultrasound images202-203 to the training algorithm step 204. Reducing the scale ofultrasound images 202-203 as a preprocessing step may reduce the amountof image data to be processed during the training act 204, and thus mayreduce the corresponding computing resources required for the trainingact 204 and/or improve the speed of the training act 204.

Various additional or alternative pre-processing acts may be performedin act 201. For example, these acts may include data normalization toensure that the various ultrasound frames 202-203 used for training havegenerally the same dimensions and parameters.

In act 204, the various inputs on the training ultrasound data 202 and203 are provided as labeled data for use in training a machine learning(ML) algorithm. For example, the various training data 202 and 203 maybe inputted into a deep neural network that can learn how to correctlypredict a gate position, gate width, and/or correction angle on newultrasound images.

The result of the training may be the AI model 206, which represents themathematical weights and/or parameters learned by the deep neuralnetwork to predict an accurate gate position, width, and/or correctionangle on new ultrasound images. The training act 204 may involve variousadditional acts (not shown) to generate a suitable AI model 206. Forexample, these various deep learning techniques include regression,classification, feature extraction, and the like. Any generated AImodels may be iteratively tested to ensure they are not overfitted andsufficiently generalized for identifying gate position, width, and/orcorrection angle on new ultrasound images. In various embodiments, themachine learning may be supervised or unsupervised.

For example, in some embodiments, once the training images 202 and 203are obtained with tracked input for gate position, width, and/orcorrection angle (e.g., the labeled data for training), a deep neuralnetwork may use them as inputs and the associated expert details of thegate position, width, and/or correction angle as desired may beoutputted to determine value sets of neural network parameters definingthe neural networks.

In some embodiments, the various user interface elements associated withthe gate position, gate width, and/or correction angle may form a maskon the underlying B-mode image. In some embodiments, the neural networkmay be configured to receive one or more ultrasound images as input andto have a softmax layer as an output layer. The output layer may specifywhether the corresponding pixels of the underlying B-mode image formpart of the user interface elements for specifying the gate location,gate width, and/or correction angle (e.g., whether the correspondingpixels form the various user interface elements 17, 18, 19, 20, 21, 22discussed above with respect to FIG. 1).

In some embodiments, the training images file may include an imageidentifier field for storing a unique identifier for identifying theunderlying B-mode image, and a segmentation mask field for storing anidentifier for specifying the user interface elements representing thegate location, gate width, and/or correction angle inputted by anoperator.

In some embodiments, using a cross-validation method on the trainingprocess would optimize neural network hyper-parameters to try to ensurethat the neural network can sufficiently learn the distribution of allpossible details for the gate position, width, and/or correction anglewithout overfitting to the training data. In some embodiments, afterfinalizing the neural network architecture, the neural network may betrained on all of the data available in the training image files.

In various embodiments, batch training may be used and each batch mayconsist of multiple images, thirty-two for example, wherein each exampleimage may be scaled to be gray-scale, 256*256 pixels, without anypreprocessing applied to it.

In some embodiments, the deep neural network parameters may be optimizedusing the Adam optimizer with hyper-parameters as suggested by Kingma,D. P., Ba, J. L.: Adam: a Method for Stochastic Optimization,International Conference on Learning Representations 2015 pp. 1-15(2015), the entire contents of which are incorporated herewith. Theweight of the convolutional layers may be initialized randomly from azero-mean Gaussian distribution. In some embodiments, the Keras™ deeplearning library with TensorFlow™ backend may be used to train and testthe models.

In some embodiments, during training, different steps may be taken tostabilize learning and prevent the model from over-fitting. Using theregularization method, e.g., adding a penalty term to the loss function,has made it possible to prevent the coefficients or weights from gettingtoo large. Another method to tackle the over-fitting problem is dropout.Dropout layers limit the co-adaptation of the feature extracting blocksby removing some random units from the neurons in the previous layer ofthe neural network based on the probability parameter of the dropoutlayer. Moreover, this approach forces the neurons to follow overallbehaviour. This implies that removing the units would result in a changein the neural network architecture in each training step. In otherwords, a dropout layer performs similar to adding random noise to hiddenlayers of the model. A dropout layer with the dropout probability of 0.5may be used after the pooling layers.

Data augmentation is another approach to prevent over-fitting and addmore transitional invariance to the model. Therefore, in someembodiments, the training images may be augmented on-the-fly whiletraining. In every mini-batch, each sample may be translatedhorizontally and vertically, rotated and/or zoomed, for example. Thepresent invention is not intended to be limited to any one particularform of data augmentation, in training the AI model. As such, any modeof data augmentation which enhances the size and quality of the dataset, and applies random transformations which do not change theappropriateness of the label assignments may be employed, including butnot limited to image flipping, rotation, translations, zooming, skewing,and elastic deformations.

Referring still to FIG. 2, after training has been completed, the setsof parameters stored in the storage memory may represent a trainedneural network for masking out the user interface elements correspondingto the gate location, size, and/or correction angle.

In order to assess the performance of the model, the stored modelparameter values can be retrieved any time to perform image assessmentthrough applying an image to the neural networks represented thereby.

In some embodiments, the deep neural network may include various layerssuch as convolutional layers, max-pooling layers, and fully connectedlayers. In some embodiments, the final layers may include a softmaxlayer as an output layer having outputs which eventually woulddemonstrate respective determinations that an input set of pixels formpart of the user interface elements corresponding to the gate location,size and/or correction angle. Accordingly, in some embodiments, theneural network may take at least one image as an input and output abinary mask indicating which pixels belong to the user interfaceelements corresponding to the gate location, size, and/or correctionangle (e.g., the AI model classifies which area each pixel belongs to).

To increase the robustness of the AI model 206, in some embodiments, abroad set of training data may be used at act 204. For example, it isdesired that ultrasound images of a plurality of anatomical features(for example a variety of arteries), both transverse and longitudinally,and at differing frequencies, depths and gains be included in thetraining ultrasound images 202 and 203.

More specifically, training medical images 202 and 203 may be labeledwith one or more features associated with/are hallmarks of an optimalgate placement. This may include identifying a variety of featuresvisualized in the captured training medical image including but notlimited to vessel walls, skin and other relevant and proximal anatomicallandmarks. In at least some embodiments, this data may be received fromtrainer/user input. For example, a trainer/user may label the featuresrelevant for the application visualized in each training image.

The image labelling can be performed, for example, by a trainer/userobserving the training ultrasound images, via a display screen of acomputing device, and manually annotating the image via a userinterface. In some aspects, the training ultrasound images used for themethod herein will only be images in which the image quality is of asufficient quality threshold to allow for proper, accurate and optimalgate placement. For example, this can include training ultrasound imageshaving a quality ranging from a minimum quality in which target featuresare just barely visible for labelling (e.g., annotating), to excellentquality images in which the target features are easily identifiable. Invarious embodiments, the training medical images can have differentdegrees of images brightness, speckle measurement and SNR. Accordingly,training ultrasound images 202 and 203 can include a graduation oftraining medical images ranging from images with just sufficient imagequality to high image quality. In this manner, the machine learningmodel may be trained to identify features on training medical imagesthat have varying levels of sufficient image quality for laterinterpretation and probability assessment.

As noted above, there are optimal angles for acquiring ultrasound magesof anatomical features such as blood vessels (hence the need for gateplacement). However, unskilled or novice ultrasound operators may nothave developed the skillset to achieve this. Thus, training AI model206, with off-angle ultrasound images may increase the robustness of themodel, so as to be operational and accurate when new ultrasound imagesare acquired by unskilled or novice operators. This is compounded by thefact that AI model 206 may be trained on a plurality of different likefeatures, with differing characteristics, in varying locations in thebody.

Overall, the scope of the invention and accorded claims are not intendedto be limited to any one particular process of training AI model 206.Such examples are provided herein by way of example only. AI model 206may be trained by both supervised and unsupervised learning approachesat 204 although due to scalability, unsupervised learning approaches,which are well known in the art, are preferred. Other approaches may beemployed to strengthen AI model 206.

For example, unique to Doppler imaging, AI model 206 may be trained witha plurality of training ultrasound frames, each of said trainingultrasound frames comprising a mask of an optimal gate (location and/orsize and/or angle) created in Doppler mode, from a plurality of manualinputs, said manual inputs defining a labeled mask of optimal gateparameters. Referring to FIG. 11, screen 8 is shown on which anultrasound image in color Doppler mode 3 is displayed. The ultrasoundimage in color Doppler mode 3 includes features of a body, such as bloodvessel walls 12, 14 and skin 16. Also displayed on the ultrasound image3 is a gate 17 that indicates where a Doppler mode signal in the tissuecorresponding to the gate location is obtained. The extent of the gate17 is defined by ends 18, 19, and the direction of the gate 17 isdefined by line 20. Gate boundary points (11) are placed around gate 17delineating the gate from other non-gate image areas (mask 40). Morespecifically, image areas that are on the right side “R” of gateboundary points 11 may be removed prior to labeling such images asacceptable (A) for training AI model 206 or 622. Likewise, image areasthat are on the left side “L” of the gate boundary points 11 that are oneach side of may be removed prior to labeling such images as acceptable(A) for training AI model 206 or 622. Further illustrated in FIG. 11, inrespect to screen 8 are exemplary user experience/interaction featuressuch as color Doppler mode visual indicator 44, freeze screen contactpoint 46, film contact point 48, photo capture contact point 50 andreturn to previous screen contact point 52. It is to be understood thatthese and any other contact points on a user interface described hereinmay be via a button, a touch-sensitive region of the user interface, adial, a slider, a drag gesture, a voice command, a keyboard, a mouse, atrackpad, a touchpad, or any combination thereof.

Referring back to FIG. 2, once a satisfactory AI model 206 is generated,the AI model 206 may be deployed for execution on a neural network 210to predict the characteristics of the gates on new ultrasound images208. Notably, the neural network 210 is shown in FIG. 2 for illustrationas a convolution neural network—which may have various nodes in theinput layer, hidden layers, and output layers. However, in variousembodiments, different arrangements of the neural network 210 may bepossible.

In various embodiments, prior to being processed for assessment ofpredicted optimal gate parameters, the new ultrasound images 208 mayoptionally be pre-processed. This is shown in FIG. 2 with thepre-processing act 207 in dotted outline. In some embodiments, thesepre-processing acts 207 may be analogous to the pre-processing acts 201performed on the training ultrasound frames 202 and 203 (e.g.,processing through a high contrast filter and/or scaling), to betteralign the new ultrasound images 208 with the training ultrasound images202 and 203, and thereby facilitate improved accuracy in predictingoptimal gate parameters. For example, pre-processing an input image mayhelp standardize the input image so that it matches the format (e.g.,having generally the same dimensions and parameters) of the trainingultrasound images 202 and 203 that the AI machine model 206 is trainedon.

In various embodiments, the new ultrasound images 208 may be live imagesacquired by an ultrasound imaging system (e.g., the system discussedwith respect to FIGS. 4 and 5 below). For example, the AI model 206 maybe deployed for execution on the scanner 412 and/or the display device 6discussed in more detail below. Additionally, or alternatively, the AImodel 206 may be executed on stored (as opposed to new) ultrasoundimages that were previously acquired (e.g., as may be stored on aPicturing Archiving and Communication System (PACS)).

Whether the images are stored ultrasound images or new ultrasound images208, the AI model 206 enables the neural network 210 to generate aprediction of optimal gate parameters (one or more of gate location,gate size and gate angle) depicted then in ultrasound image frames 212.Further illustrated in ultrasound image frames 212 are features of abody, such as blood vessel walls 228 and 230 and skin 226. Alsodisplayed on the ultrasound images 212 is a gate 214 that indicateswhere a Doppler mode signal in the tissue corresponding to the gatelocation is obtained. The extent of the gate 214 is defined by ends 216and 218, and the direction of the gate 214 is defined by line 220.Correction lines 222 and 224 are shown positioned parallel to the walls228 and 230 of the blood vessel being scanned.

When executed in this manner, the AI model 206 may allow the neuralnetwork 210 to predict the position, size, and/or correction angle ofgate to be placed on the new ultrasound frames 208, resulting incorresponding ultrasound frames 212 with a predicted position, size,and/or correction of the gate. The predicted characteristics of the gatemay then be used for input to the ultrasound scanner to acquire Dopplersignals. For example, the predicted characteristics of the gate may beinterpreted by the ultrasound system as if the predicted gate location,size, and/or correction angle were manually-inputted user interfacemanipulations of an operator.

In some embodiments, the ultrasound system may be configured to applythe AI model 206 periodically (e.g., in regular intervals of between 1-4seconds) to automatically optimally position and update position forgate position, size and/or correction angle.

An ultrasound scanner may generally transmit and receive ultrasoundsignals according to an ultrasound sequence when generating the liveultrasound image feed (e.g., the sequence and characteristics in whichultrasound pulses are directed to the tissue and the resultant echosignals received). Manual modification of the gate characteristicsgenerally results in an update of the ultrasound sequence used toacquire ultrasound signals.

However, gate characteristics predicted by the AI model 206 may notalways be suitable for updating the ultrasound sequence. As such, insome embodiments, the sequence may only be updated if the output of theAI model 206 is of a high enough confidence level (e.g., 70%) and/or ifone or more of the gate position, size, and/or correction angle haschanged (e.g., moved or adjusted) beyond a threshold amount (e.g.,10-40%). This may reduce the constant updating of the ultrasoundsequence that might otherwise occur, and the resulting associatedjumping or flickering in the ultrasound images being displayed.

Referring still to FIG. 2, in some embodiments, the ultrasound frames212 with predicted gate characteristics may optionally themselves beused for training and/or reinforcing the AI model 206. This is shown inFIG. 2 with a dotted line from the ultrasound frames 212 with thepredicted gate characteristics being provided to the training act 204.For example, the transmission of such reinforcement training ultrasounddata may be sent to a centralized server 520 which collects variouspredicted gate characteristics for updating the AI model 206, using thedistributed network 500 discussed in greater detail below with respectto FIG. 5.

Referring to FIG. 3, shown there is a diagram of a method for trainingand deployment of an AI model for placement of a color box duringultrasound imaging, according to an embodiment of the present invention.The elements of FIG. 3 generally correspond to those in FIG. 2, exceptinstead of training an AI model 206 for placement of gate position,size, and/or correction angle, FIG. 3 is directed to training an AImodel 306 for placement of a color box. The techniques, processes, andvariations discussed above with respect to FIG. 2 may thus generally beapplicable to FIG. 3 also.

Generally, when an ultrasound system is configured to be in ColorDoppler mode, the operator is required to place a color box in the userinterface for identifying the corresponding tissue in which the colormode signal is obtained and displayed. For example, the color box may beshown as an angled box (e.g., parallelogram), as is illustrated in theultrasound images 302 in FIG. 3.

The training ultrasound frames (302-303), which as above in regard toFIG. 2 may be B-mode or Doppler images, includes ultrasound frameslabeled as Acceptable with color box placements that are tagged asacceptable and representative of an optimal color box location and sizeand may include ultrasound frames labeled as Unacceptable that aretagged respectively as unacceptable and unrepresentative of an optimalcolor box gate location and size. Both the training ultrasound frameslabeled as Acceptable and Unacceptable may themselves be used fortraining and/or reinforcing AI model 306. This is shown in FIG. 3 withtracking lines from ultrasound images labeled as both Acceptable andUnacceptable, to training algorithm step 304.

In some embodiments, an optional pre-processing act 301 may be performedon the underlying ultrasound image frames 302 and 303 to facilitateimproved performance and/or accuracy when training the machine learning(ML) algorithm. For example, it may be possible to pre-process theultrasound images 302 and 303 through a high contrast filter to reducethe granularity of greyscale on the ultrasound images 302 and 303.

Additionally, or alternatively, it may be possible to reduce scale ofthe ultrasound images 302-303 prior to providing the ultrasound images302-303 to the training algorithm step 304. Reducing the scale ofultrasound images 302-303 as a preprocessing step may reduce the amountof image data to be processed during the training act 304, and thus mayreduce the corresponding computing resources required for the trainingact 304 and/or improve the speed of the training act 304.

Various additional or alternative pre-processing acts may be performedin act 301. For example, these acts may include data normalization toensure that the various ultrasound frames 302-303 used for training havegenerally the same dimensions and parameters.

In act 304, the various inputs on the training ultrasound data 302 and303 are provided as labeled data for use in training a machine learning(ML) algorithm. For example, the various training data 302 and 303 maybe inputted into a deep neural network 310 that can learn how tocorrectly predict a color box placement/location and size on newultrasound images of vascular features as shown in ultrasound images312.

As such, once the ML algorithm is trained using the various techniquesdiscussed above, an AI model 306 may be developed and the AI model 306may be deployed into a neural network 310. When new ultrasound images308 (optionally preprocessed at act 307) are fed into the neural network310 configured with AI model 306, it may be able to predict the optimalposition for the placement of the color box as shown on ultrasoundimages 312.

Further illustrated in ultrasound image frames 312 in FIG. 3 arefeatures of a body, such as blood vessel walls 328 and 330 and skin 326.Also displayed on the ultrasound images 312 is a color box 335, alongwith striated markers of the insonation angle 336. This the anglebetween the path of the Doppler pulses and the direction of flow in thevascular feature as indicated by the orientation of the color box. Asnoted above, when this angle is 90° (top), there will be no frequencyshift because)cos(90°=0. With angle correction in any vascular feature,more of the flow becomes detectable.

The present embodiments may be deployed in various example scenarios.Also, variations to the embodiments described herein may be possible.

For example, as discussed herein, the AI model 206 for predicting gatelocation/size/correction angle (FIG. 2) and the AI model 306 forpredicting color box location (FIG. 3) have been described as separateAI models. However, in various embodiments, the two may be combined intoa single AI model that takes both the training inputs 202, 302 discussedin FIGS. 2 and 3 and generates a single AI model for predicting bothgate size/location/correction angle and color box location. Anillustration of this single but combined function AI model isillustrated more fully in FIG. 6.

Referring to FIG. 6, shown there is a diagram of a method 600 fortraining and deployment of a dual AI model for i) placement of gate(location, size, and/or correction angle); and ii) placement of a colorbox during ultrasound imaging of a vascular feature, according to anembodiment of the present invention. The elements of FIG. 6 generallycorrespond to those respectively in FIGS. 2 and 3, except instead oftraining an AI model 206/306 for singular placement of gate position,size, and/or correction angle or color box, FIG. 6 is directed totraining a dual function AI model 622 for both placement of gate(location, size, and/or correction angle) and placement of a color boxduring ultrasound imaging of a vascular feature. The techniques,processes, and variations discussed above with respect to FIGS. 2 and 3may thus generally be applicable to FIG. 6 also.

The training ultrasound frames (612, 614, 616 and 618), which as aboveregarding FIGS. 2 and 3 may be B-mode ultrasound images or Doppler modeimages, include ultrasound frames labeled as Acceptable (for example612) with gate parameters that are tagged as acceptable andrepresentative of an optimal gate location and/or size and/or angle andultrasound frames labeled as Unacceptable (for example 614) that aretagged respectively as unacceptable and unrepresentative of an optimalgate location and/or size and/or angle. Ultrasound frames labeled asAcceptable (for example 616) with color box location and size which aretagged as acceptable and representative of an optimal color box locationand size and ultrasound frames labeled as Unacceptable (for example 614)with color box location and size which are tagged respectively asunacceptable and unrepresentative of an optimal color box location andsize. Both the training ultrasound frames labeled as Acceptable andUnacceptable may themselves be used for training and/or reinforcingDoppler gate and color box collective AI model 622. This is shown inFIG. 6 with tracking lines from ultrasound images labeled as bothAcceptable and Unacceptable, to training algorithm step 620.

In some embodiments, an optional pre-processing act 610 may be performedon the underlying ultrasound image frames 612-618 to facilitate improvedperformance and/or accuracy when training the machine learning (ML)algorithm. For example, it may be possible to pre-process the ultrasoundimages 612-618 through a high contrast filter to reduce the granularityof greyscale on the ultrasound images 612-618.

Additionally, or alternatively, it may be possible to reduce scale ofthe ultrasound images 302-303 prior to providing the ultrasound images612-618 to the training algorithm step 620. Reducing the scale ofultrasound images 612-618 as a preprocessing step may reduce the amountof image data to be processed during the training act 620, and thus mayreduce the corresponding computing resources required for the trainingact 620 and/or improve the speed of the training act 620.

Various additional or alternative pre-processing acts may be performedin act 620. For example, these acts may include data normalization toensure that the various ultrasound frames 612-618 used for training havegenerally the same dimensions and parameters.

In act 620, the various inputs on the training ultrasound data 612-618are provided as labeled data for use in training a machine learning (ML)algorithm. For example, the various training data 612-618 may beinputted into a deep neural network 624 that can learn how to correctlypredict both optimal gate parameters and optimal color boxplacement/location and size on new ultrasound images of vascularfeatures as shown in ultrasound images 630 (AI model predicted placementof gate alone), 632 (AI model predicted placement of color box alone),and 634 (AI model predicted placement of both optimal gate parametersand optimal color box placement/location and size.

As such, once the ML algorithm is trained using the various techniquesdiscussed above, an AI model 622 may be developed and the AI model 622may be deployed into a neural network 624. When new ultrasound images628 and 630, for example, (optionally preprocessed at act 626) are fedinto the neural network 624 configured with AI model 622, it may be ableto predict i) optimal gate parameters (image 630); ii) optimal color boxplacement/location and size (image 632) and iii) both optimal gateparameters and optimal color box placement/location and size (image634).

Further illustrated in ultrasound image frames 630-634 in FIG. 6 arefeatures of a body, such as blood vessel walls 640 and 648 and skin 646.Also displayed on the ultrasound images 630 and 634 is a gate 644 thatindicates where a Doppler mode signal in the tissue corresponding to thegate location is obtained. The extent of the gate 644 is defined by ends647 and 650, and the direction of the gate 644 is defined by line 651.Correction lines 652 and 654 are shown positioned parallel to the walls640 and 648 of the blood vessel being scanned. Also displayed on theultrasound images 632 and 634 is a color box 656 along with striatedmarkers of the insonation angle 658.

In various embodiments, the embodiments described above for predictingcolor box may be used in Color Doppler modes and/or Power Doppler modes.

In various embodiments, the placement of a gate may be performed ineither mono mode (e.g., mono PW Doppler mode, where only PW samplingsare obtained and the B-mode is turned off but B-mode can be manuallytoggled on by the operator when desired); or duplex mode (where multipletypes of ultrasound signals are interleaved to generate a liveultrasound image with multiple types of data together). An example ofduplex scanning is B-mode ultrasound signals and PW ultrasound signalsbeing interleaved together to provide a live B-mode image of structurebeing imaged and also PW Doppler data where the gate is placed. Theembodiments of FIG. 2 may be employed in this instance. Another exampleof duplex scanning is B-mode/Color Doppler, where the embodiments ofFIG. 3 may be employed.

Additionally or alternatively, FIG. 2 has generally been illustrated inrelation to placement of a gate on a simple B-mode image with no othermodes activated, and FIG. 3 has similarly been illustrated in relationto placement of a color box on a simple B-mode image with no other modesactivated. However, in some embodiments, both a color box and a gate maybe present on a B-mode image. This may be the case in certain ultrasoundsystems that allow for three different imaging modes to be combinedtogether. Examples of these include adding a further mode to the duplexB/PW combination already discussed: e.g., adding a Color Doppler orPower Doppler mode. This results in additional combination modes thatcombine: B-mode, PW Doppler, and Color Doppler together (B/PW/Color); orB-mode, PW Doppler, and Power Doppler together (B/PW/Power). In thesetriplex modes, the AI model(s) discussed herein may be used together topredict both the gate characteristics for PW Doppler, and a color boxfor Color Doppler or Power Doppler.

Moreover, while the placement of a gate in relation to FIG. 2 above hasgenerally been in relation to a spectral Doppler mode, in variousembodiments, it may be possible to use the embodiments of FIG. 2 inimaging modes that do not involve spectral Doppler at all. For example,the embodiments described herein may generally be used to predict gatecharacteristics (e.g., location). As such, the present embodiments maybe employed in any ultrasound imaging mode that traditionally requiresmanual input of a gate.

For example, M-Mode (e.g., Motion-mode) provides a time motion displayof ultrasound signals along a chosen ultrasound line. The embodiments ofFIG. 2 may be employed to learn and predict the positioning of a gate inM-mode.

In another example, some ultrasound systems have elastography modes thatprovide a map and/or measurement of tissue stiffness (e.g., using shearwave vibrations and/or acoustic radiation force imaging (ARFI)). Inthese modes, a region of tissue may be continuously measured withoutputs being displayed in real-time to the user. In order to achieveproper targeting within the tissue, it may be helpful to provide a“gate” to position the desired elastography measurement area. Thesystems and methods described herein may be used in these elastographymodes to predict positioning of the gate for live elastographymeasurements.

Referring to FIG. 4, an exemplary system 400 on which the presentembodiments may be practised is shown. The system 400 may includeultrasound scanner 412 (referred to herein as “scanner” for brevity)that acquires the ultrasound imaging frames 2, 202, 302 (shown in FIGS.1, 2, and 3 above) and which the placement of the gate or color boxwould result in acquisition of corresponding ultrasound signals (e.g.,Doppler signals). The system 400 may include an ultrasound scanner 412with a processor 414, which may be connected to a non-transitorycomputer readable memory 416 storing computer readable instructions 418,which, when executed by the processor 414, may cause the scanner 412 toprovide one or more of the functions of the system 400. Such functionsmay be, for example, the acquisition of ultrasound data, the processingof ultrasound data, the scan conversion of ultrasound data, thetransmission of ultrasound data or images to a display device 6, thedetection of operator inputs to the scanner 412, and/or the switching ofthe settings of the scanner 412 (e.g., to update the ultrasound sequenceto acquire Doppler signals in area represented by the gate position,gate width, correction angle and/or color box, as predicted by AI models206, 306).

Also stored in the computer readable memory 416 may be computer readabledata 420, which may be used by the processor 414 in conjunction with thecomputer readable instructions 418 to provide the functions of thesystem 400. Computer readable data 420 may include, for example,configuration settings for the scanner 412, such as presets thatinstruct the processor 414 how to acquire Doppler signals in the areacorresponding to the AI-predicted gate position, gate width, correctionangle and/or color box.

The scanner 412 may include a communications module 422 connected to theprocessor 414. In the illustrated example, the communications module 422may wirelessly transmit signals to and receives signals from the displaydevice 6 along wireless communication link 424. The protocol used forcommunications between the scanner 412 and the display device 6 may beWiFi™ or Bluetooth™, for example, or any other suitable two-way radiocommunications protocol. The scanner 412 may operate as a WiFi™ hotspot,for example. Communication link 424 may use any suitable wirelessnetwork connection. In some embodiments, the communication link betweenthe scanner 412 and the display device 6 may be wired. For example, thescanner 412 may be attached to a cord that may be pluggable into aphysical port of the display device 6.

The display device 6 may be, for example, a laptop computer, a tabletcomputer, a desktop computer, a smart phone, a smart watch, spectacleswith a built-in display, a television, a bespoke display or any otherdisplay device that is capable of being communicably connected to thescanner 412. The display device 6 may host a screen 8 and may include aprocessor 434, which may be connected to a non-transitory computerreadable memory 436 storing computer readable instructions 438, which,when executed by the processor 434, cause the display device 6 toprovide one or more of the functions of the system 400. Such functionsmay be, for example, the receiving of ultrasound data that may or maynot be pre-processed; scan conversion of ultrasound data that isreceived into a ultrasound images; processing of ultrasound data inimage data frames; the display of an ultrasound image on the screen 8;the display of a user interface elements; the control of the scanner412; executing the software module that runs parallel to the standardultrasound imaging software for tracking operator inputs of userinterface elements related to the gate and/or color box; the storage,application, deployment, reinforcing and/or training of an AI model withrespect to the placement of a gate and/or color box; and/or adjusting ofultrasound settings to correspond to the placement of the gate and/orcolor box.

Also stored in the computer readable memory 436 may be computer readabledata 440, which may be used by the processor 434 in conjunction with thecomputer readable instructions 438 to provide the functions of thesystem 400. Computer readable data 440 may include, for example,settings for the scanner 412, such as presets for acquiring Dopplerframes based on the AI-predicted user interface elements of the gateand/or color box; ultrasound data received from the scanner 412;settings for a user interface displayed on the screen 8; and/or one ormore AI models. Settings may also include any other data that isspecific to the way that the scanner 412 operates or that the displaydevice 6 operates.

It can therefore be understood that the computer readable instructionsand data used for controlling the system 400 may be located either inthe computer readable memory 416 of the scanner 412, the computerreadable memory 436 of the display device 6, and/or both the computerreadable memories 416, 436.

The display device 6 may also include a communications module 442connected to the processor 434 for facilitating communication with thescanner 412. In the illustrated example, the communications module 442wirelessly transmits signals to and receives signals from the scanner412 on wireless communication link 424. However, as noted, in someembodiments, the connection between scanner 412 and display device 6 maybe wired.

Referring to FIG. 5, a distributed network 500 is shown in which thereare multiple similar or different scanners 412, 412 a, 412 b connectedto their corresponding display devices 6, 6 a, 6 b and either connecteddirectly, or indirectly via the display devices, to a communicationsnetwork 510, such as the internet. The scanners 412, 412 a, 412 b may beconnected onwards via the communications network 510 to a server 520.

The server 520 may include a processor 522, which may be connected to anon-transitory computer readable memory 524 storing computer readableinstructions 526, which, when executed by the processor 522, cause theserver 520 to provide one or more of the functions of the distributednetwork 500. Such functions may be, for example, the receiving ofultrasound data that may or may not be pre-processed, the scanconversion of ultrasound data that is received into an ultrasound image,the processing of ultrasound data in image data frames, the control ofthe scanners 412, 412 a, 412 b, and/or machine learning activitiesrelated to one or more AI models 206, 306 (as shown in FIGS. 2 and 3).Such machine learning activities may include the training and/orreinforcing of one or more AI models 206, 306.

Also stored in the computer readable memory 524 may be computer readabledata 528, which may be used by the processor 522 in conjunction with thecomputer readable instructions 526 to provide the functions of thedistributed network 500. Computer readable data 528 may include, forexample, settings for the scanners 412, 412 a, 412 b such as presetparameters for acquiring ultrasound data, settings for user interfacesdisplayed on the display devices 6, 6 a, 6 b, and one or more AI models206, 306. For example, one AI model may be an AI model for predictinglocations for placement, width, and correction angle of gates and/orcolor boxes for Doppler signals used by the scanners 412, 412 a, 412 b.Settings may also include any other data that is specific to the waythat the scanners 412, 412 a, 412 b operate or that the display devices6, 6 a, 6 b operate.

It can therefore be understood that the computer readable instructionsand data used for controlling the distributed network 500 may be locatedeither in the computer readable memory of the scanners 412, 412 a, 412b, the computer readable memory of the display devices 6, 6 a, 6 b, thecomputer readable memory 524 of the server 520, or any combination ofthe foregoing locations.

As the scanners 412, 412 a, 412 b and corresponding display devices 6, 6a, 6 b may be different, the placement, sizing, and correction angle ofthe gates and/or the positioning of the color box (e.g., when usingDoppler related modes) would generally be performed by differentoperators. This may allow the various inputs for training the AI models206, 306 (as shown in FIGS. 2 and 3) to be more robust, since it isbased on different user input. These various datasets can then beconsolidated for training and/or reinforcing the AI models 206, 306 atthe server 520. As the AI models 206, 306 are updated from time to timebased on new training data collected at the various scanners 412, 412 a,412 b and corresponding display devices 6, 6 a, 6 b, in someembodiments, AI models 206, 306 used at the display devices 6, 6 a, 6 bmay be updated from the AI models 206, 306 present in the server 520.

FIG. 7 is flowchart diagram of the steps for training the AI model ofFIGS. 2 and 6, according to an embodiment of the present invention.Method 700 is described below with regard to the systems and componentsdepicted in FIGS. 4 and 5, though it should be appreciated that method700 may be implemented with other systems and components withoutdeparting from the scope of the present disclosure. In some embodiments,method 700 may be implemented as executable instructions in anyappropriate combination of the imaging system 400, for example, anexternal computing device connected to the imaging system 400, incommunication with the imaging system 400, and so on. As one example,method 700 may be implemented in non-transitory memory of a computingdevice, such as the controller (e.g., processor) of the imaging system400 in FIG. 4. At 712, method 700 may include acquiring a dataset ofsample images for training the neural network. Each sample image in thedataset may be a sample ultrasound image depicting an anatomicalfeature, for example a vascular feature (and more specifically here, forexample, an artery).

Referring still to FIG. 7, in step 712, a training ultrasound image maybe obtained. For example, a training ultrasound image may be acquired bythe scanner 412 (as shown in FIG. 5) transmitting and receivingultrasound energy. The training ultrasound image may be a post-scanconverted ultrasound image. However, as discussed herein, it is to bemade clear that the AI models of the present invention may be trained byB-mode ultrasound images and/or Doppler ultrasound images, withoutlimitation. Furthermore, while the method of FIG. 7 is described inrelation to a single training ultrasound image, the method may alsoapply to the use of multiple training ultrasound images. While themethod of FIG. 7 is described in relation to a post-scan ultrasoundimage, it is to be understood that pre-scan images, may be used, asdescribed in U.S. patent application Ser. No. 17/187,851 filed Feb. 28,2021, the entire contents of which are incorporated herein by reference.

Optionally, in step 714 (as shown in dotted outline), the resolution ofthe training ultrasound image may be adjusted. For example, theresolution may be increased or decreased. The purpose of this may be toprovide the labeler (e.g., a medical professional with relevant clinicalexpertise) with training ultrasound images that have a more standardizedappearance. This may help to maintain a higher consistency with whichthe labeler identifies vascular features in the training ultrasoundimages. Besides the resolution, other parameters of the trainingultrasound image may also be adjusted such as input scaling, screensize, pixel size, aspect ratio, and the removal of dead space, asdescribed above (including, for example, data augmentation and otherpreprocessing steps).

In step 716, the training ultrasound image may be displayed on a displaydevice, such as the display device 6 discussed above in relation to FIG.5 and input is received to apply and adjust gate on the vascularfeature. As the gate defines the size and location of the area fromwhich Doppler information is obtained and it is delineated as a pair ofcrosshairs within the 2D image, an optimally trained gate, for any, forexample, vascular feature, should be as small as possible to excludeerroneous signal arising from adjacent vessels or marginal flow. Toolarge a gate may admit erroneous signals from adjacent vessels and toosmall a gate may give the false impression of reduced or even absentflow. However, a smaller gate also reduces computation time andincreases the frame rate, thereby allowing more accurate depiction offlow. To maximize depiction of flow, the gate should be positioned overthe central part of the vascular feature in the ultrasound image. Atstep 716, such adjustments may be made as part of employing a pluralityof ultrasound images to train the AI model.

At step 718, input is received to switch to Doppler signal mode.Optionally, at step 720 thereafter, each image may be optimized using,for example, flow detection, adjustments to color gain, adjustments towall filters etc.

At step 722, the labeler can then identify a confirmation of optimalgate parameters. The labeler then can mark the training ultrasound imagearound the optimal gate (location, size and angle) that the labeler hasidentified in the training ultrasound image. In step 722, the systemthat is used for the training may receive the identification ofoptimized gate parameters and such system may generate, from thelabeler's marking inputs, a labeled training ultrasound image, anddisplay it on the display device.

Once the training ultrasound image has been marked and labeled, thesystem may then remove, optionally, regions of the labeled ultrasounddata frame that are both outside the area of optimized gate and outsideareas relevant for the AI model to recognize a placement location forthe gate. For example, the labeled ultrasound data frame may betruncated at one or more sides. Truncation of some of the ultrasounddata may allow the training of the AI model to proceed more quickly. Atstep 724, there is provided a redirection to complete steps 712-722 aplurality of times, for i) additional images from the same ultrasounddevice; ii) additional data from other ultrasound devices; and/or iii)additional images as acquired from multiple users, thereby to build arobust gate placement specific AI model. At step 726, the labeled rawultrasound data frame is then used for training the AI model 206. Atstep 728, once training is completed, the AI model may be used toperform predictions on an unseen dataset to validate its performance,such evaluation at step 728 feeding data back to train the AI model atstep 726.

As described herein, the AI models of the present invention may furtherbe trained with image data conveyed from a cloud-based storage (e.g.,previous exams stored on the cloud-based storage that may have indicatedoptimal gate placement. Further, the the AI models of the presentinvention may further be trained as described in FIG. 11.

FIG. 8 is flowchart diagram of the steps for training the AI model ofFIGS. 3 and 6, according to an embodiment of the present invention.Method 800 is described below with regard to the systems and componentsdepicted in FIGS. 4 and 5, though it should be appreciated that method800 may be implemented with other systems and components withoutdeparting from the scope of the present disclosure. In some embodiments,method 800 may be implemented as executable instructions in anyappropriate combination of the imaging system 400, for example, anexternal computing device connected to the imaging system 400, incommunication with the imaging system 400, and so on. As one example,method 800 may be implemented in non-transitory memory of a computingdevice, such as the controller (e.g., processor) of the imaging system400 in FIG. 4. At 812, method 800 may include acquiring a dataset ofsample images for training the neural network. Each sample image in thedataset may be a sample ultrasound image depicting for example, avascular feature, more specifically, for example an artery.

Referring still to FIG. 8, in step 812, a training ultrasound image maybe obtained. For example, a training ultrasound image may be acquired bythe scanner 412 (as shown in FIG. 5) transmitting and receivingultrasound energy. The training ultrasound image may be a post-scanconverted ultrasound image. However, as discussed herein, it is to bemade clear that the AI models of the present invention may be trained byB-mode ultrasound images and/or Doppler ultrasound images, withoutlimitation. Furthermore, while the method of FIG. 8 is described inrelation to a single training ultrasound image, the method may alsoapply to the use of multiple training ultrasound images. While themethod of FIG. 8 is described in relation to a post-scan ultrasoundimage, it is to be understood that pre-scan images, may be used, asdescribed in U.S. patent application Ser. No. 17/187,851 filed Feb. 28,2021, the entire contents of which are incorporated herein by reference.

Optionally, in step 814 (as shown in dotted outline), the resolution ofthe training ultrasound image may be adjusted. For example, theresolution may be increased or decreased. The purpose of this may be toprovide the labeler (e.g., a medical professional with relevant clinicalexpertise) with training ultrasound images that have a more standardizedappearance. This may help to maintain a higher consistency with whichthe labeler identifies vascular features in the training ultrasoundimages. Besides the resolution, other parameters of the trainingultrasound image may also be adjusted such as input scaling, screensize, pixel size, aspect ratio, and the removal of dead space, asdescribed above (including, for example, data augmentation and otherpreprocessing steps).

In step 816, the training ultrasound image may be displayed on a displaydevice, such as the display device 6 discussed above in relation to FIG.5 and input is received to apply and adjust a color box on the vascularfeature. The size, shape, and location of the color box are adjustableand define the volume of tissue from which color data are acquired. Allvelocity information from this defined volume of tissue is presented ascolor-encoded Doppler shifts in the image field. The frame ratedecreases as box size increases, so image resolution and quality areaffected by box size and width. Increasing the size or width will reducethe frame rate and increase the required processing power and time.Thus, in training an AI model for color box placement, adjustments atstep 816 may be made accordingly. The color box overlay should be assmall and superficial as possible while still providing the necessaryinformation, thereby maximizing the sampling or frame rate. Too big anoverlay reduces the frame rate and thus results in inferior depiction offlow. At step 816, all such adjustments may be made as part of employinga plurality of ultrasound images to train the AI model.

At step 818, input is received to switch to Doppler signal mode.Optionally, at step 820 thereafter, each image may be optimized using,for example, flow detection, adjustments to color gain, adjustments towall filters etc. . . .

At step 822, the labeler can then identify a confirmation of optimalcolor box location and size parameters. The labeler then can mark thetraining ultrasound image around the optimal color box location and sizethat the labeler has identified in the training ultrasound image. Instep 822, the system that is used for the training may receive theidentification of optimized color box location and size and such systemmay generate, from the labeler's marking inputs, a labeled trainingultrasound image, and display it on the display device.

Once the training ultrasound image has been marked and labeled, thesystem may then remove, optionally, regions of the labeled ultrasounddata frame that are both outside the area of optimized color box andoutside areas relevant for the AI model to recognize a placementlocation for the color box. For example, the labeled ultrasound dataframe may be truncated at one or more sides. Truncation of some of theultrasound data may allow the training of the AI model to proceed morequickly. At step 824, there is provided a redirection to complete steps812-822 a plurality of times, for i) additional images from the sameultrasound device; ii) additional data from other ultrasound devices;and/or iii) additional images as acquired from multiple users, therebyto build a robust color box placement specific AI model. At step 826,the labeled raw ultrasound data frame is then used for training the AImodel 306. At step 828, once training is completed, the AI model may beused to perform predictions on an unseen dataset to validate itsperformance, such evaluation at step 828 feeding data back to train theAI model at step 826.

Referring to FIG. 9, a flowchart diagram of a method, generallyindicated at 900, of new image acquisition of an anatomical feature,processing against an AI model and prediction of placement of optimalgate parameters, according to at least one embodiment of the presentinvention is shown. Method 900 further supports and aligns with elements204, 210, 208, 207 and 212 described above in FIG. 2. At 910, theultrasound imaging system (referred to in FIG. 4), may acquireultrasound imaging data. For example, a medical professional may operatean ultrasound scanner (hereinafter “scanner”, “probe”, or “transducer”for brevity) to capture images of a patient (whether human or animal).The ultrasound frames (B-mode) may be acquired by acquiring a series ofa images (with a frame each containing a sequence of transmitted andreceived ultrasound signals) of different views of a vascular feature.

Further, at step 910, new ultrasound imaging data may optionally bepre-processed and/or augmented as described above. At step 914, AI model206 (FIG. 2) or collective AI model 622 (FIG. 6) generates a predictionof one or more of: i) optimal gate location; ii) optimal gate size; andiii) optimal gate angle to be applied to the feature imaged in the newultrasound imaging data. Thereafter, at step 916, such predictedoptimized gate is placed on the feature imaged in the new ultrasoundimaging data and, at step 918, Spectral Doppler processing is employed.

FIG. 10 is flowchart diagram of the method steps (shown generally as950) for updating optimal gate parameters after interruption of Dopplersignal acquisition, according to at least one embodiment of the presentinvention. At 952, the ultrasound imaging system (referred to in FIG.4), may acquire ultrasound imaging data. These ultrasound frames may beacquired by acquiring a series of a images (with a frame each containinga sequence of transmitted and received ultrasound signals) of differentviews of, for example, a vascular feature. Further, at step 952, theultrasound imaging data may optionally be pre-processed and/or augmentedas described above. At step 954, AI model 206 (FIG. 2) or collective AImodel 622 (FIG. 6) may generate a prediction of one or more of: i)optimal gate location; ii) optimal gate size; and iii) optimal gateangle to be applied to the anatomical feature imaged in the B-modeultrasound imaging data. Thereafter, a predicted optimized gate isplaced on the anatomical feature imaged in the ultrasound imaging dataand, the system may receive an input at step 956 that causes it to enterspectral Doppler mode. In some instances, the acquisition of B-modeimage frames stops, and the B-mode image feed is frozen, with theresultant B-mode image on a user interface being reduced in size toaccommodate the spectral Doppler mode display portion (for example, 23in FIGS. 12-14). In some instances, it is not possible to operate induplex mode due to the particular anatomical feature being imaged, ordue to physics or due to the pulse repetition frequency (PRF). In theseand other instances, only the spectral Doppler mode display portion maybe viewable on the user interface.

Regardless, it may be desirable to be able to update gate placementusing the AI models of the present invention. In one aspect, the gateplacement may be updated by user intervention, providing a direction tothe processor to enable and engage updating steps, as described herein.In yet another embodiment, the gate may be updated (without userintervention) in response to a deficiency in the Spectral signal, theprocessor detecting such deficiency and automatically directing a gateplacement reset, using the AI models of the present invention. Suchdeficiency may indicate that a readjustment of the gate is required foroptimal signal processing in the spectral signal.

Referring still to FIG. 10, once spectral Doppler mode is entered atstep 956, a live Doppler signal is acquired at step 960 and displayed ona user interface for viewing allowing the detected Doppler shifts (kHz)to be translated into velocities (m/s). The flow is displayed as aspectral waveform of changes in velocity wherein the flow velocity isthe vertical axis with the time on the horizontal axis (see FIGS.12-14). At step 962, there is a received an input to update the gate,which triggers a temporary interruption of the Doppler signalacquisition at step 964 and at step 966, a display on the userinterface, of a captured a two-dimensional (2D) ultrasound image(“captured image”). At step 968, the AI model is applied to the capturedimage generate a prediction of one or more of an optimal updated gateposition, size and angle (the “updated optimized gate”) and the updatedoptimized gate is employed to enable corresponding SD-mode signals,therein to re-enter Doppler mode at back at step 956.

Referring to FIGS. 12-14, an exemplary sequence of displays on thetouchscreen 8 is shown as the method of FIG. 10 is performed. Prior tothe screenshot of FIG. 12, the system may be operating in a conventionalB-mode operation (act 952 of FIG. 10). The system may then receive inputto switch to a spectral Doppler mode (e.g., through navigation of a menuor some other user interface mechanism). In FIG. 12, the system 400 isin the spectral Doppler mode (act 960 of FIG. 10). The gate 17 is shownsuperimposed on the frozen B-mode image (act 954 of FIG. 10). Thespectrum 24, in the Doppler mode display portion 23, is shown in a boldline as it is live and the live Doppler signal is being acquired (act960 of FIG. 10).

In FIG. 13, the system 400 has received an input to temporarilyinterrupt the live Doppler signal for the purpose of an AImodel-directed gate update (acts 962 and 964 of FIG. 10). Such input maybe via Update Gate button 35 although as described herein, other inputoptions are fully contemplated within the scope of the invention,including auto-updates by the processor in response of Spectral signaldeficiency or interruption. A user/operator may have initiated thisaction for various reasons, for example, because the body being scannedhas moved, the ultrasound scanner 412 has moved, and/or the spectrum 24has changed in its displayed characteristics (e.g., diminished inamplitude). The mono, Doppler signal acquisition may then be interrupted(act 964 of FIG. 10) to allow for capture and display of a B-mode image(at act 966 of FIG. 10). Correspondingly, live synchronous B-mode images10 may be displayed (this is shown as the B-mode image 10 having bolddashed lines in FIG. 13). The gate 17 is shown superimposed on theB-mode image 10 in the same relative position in the display area 103 asit was in FIG. 12 (pre-gate update). However, in the example screenshotof FIG. 13, it can be seen that the vessel on which the gate 17 wascentered in FIG. 12 has now shifted in the live ultrasound image 10 sothat the gate 17 is no longer in the optimal centered position. As aresult, the gate 17 is no longer set in the originally desired locationand orientation, and the Doppler mode signal that was being acquiredimmediately prior to switching to the B-mode was therefore likely to beless accurate than it could have been.

Referring still to FIG. 13, it can also be seen that the spectrum 24 inthe Doppler mode display portion 23 is no longer acquiring a strong livesignal (as shown in a light line in FIG. 13 in contrast to the bold lineshowing the live spectrum in FIG. 12). A weaker portion 25 of thespectrum 24 in FIG. 13 also shows that the acquisition of the Dopplersignal has been fully or partially interrupted.

Act 968 of FIG. 10 provides for the acquisition of an AI model directedupdated gate (one or more of: location, size and angle). System 400continues to acquire B-mode frames and the B-mode image 10 remains live.Also, the position of the gate 17 (as shown in solid lines) has beenadjusted (without any further user intervention) by AI model 206: fromits original position relative to the display area 103 to an updatedposition (FIG. 14). Within the scope of the invention and this AI modeldirected gate updating, there is very little to no interruption to thespectral Doppler mode, as can be seen at 25 in FIGS. 13 and 36 in FIG.14 (i.e. virtually no flatlining and quickly recovered amplitude). Thisbenefit is achieved by the combination of the trained AI model foroptimal gate placement and the expeditious B-mode image in response to agate update request/input.

In FIG. 14, the gate has been updated using the AI model, the system 400returns to the Doppler mode (act 956 of FIG. 10), and the B-mode imagefeed (on the user interface) is once again frozen. In the examplescreenshot of FIG. 14, this is shown with the B-mode image havinglighter, dotted lines similar to FIG. 12. The updated gate 17 issuperimposed on the B-mode image 10. In the Doppler mode display portion23, the Doppler spectrum 24 reverts to a full live signal (act 960 ofFIG. 10).

FIG. 15 is an exemplary display on touchscreen 8, showing frozen B-modeimage 2, blood vessel walls 12 and 14, skin 16 over which AI model ofthe invention has placed color box 5 and gate 17 (comprising ends 18 and19, and direction line 20). On screen 8 are exemplary userexperience/interaction features such as freeze screen contact point 46,film contact point 48, photo capture contact point 50, along with adrop-down menu comprising a plurality of options including: B-mode 56,Color Doppler 58, Power Doppler 60, M-Mode 62, PW Doppler 64, NeedleEnhance 66, Elastography 68, and RF Mode 70.

While a number of exemplary aspects and embodiments have been discussedabove, those of skill in the art will recognize that may be certainmodifications, permutations, additions and sub-combinations thereof.While the above description contains many details of exampleembodiments, these should not be construed as essential limitations onthe scope of any embodiment. Many other ramifications and variations arepossible within the teachings of the various embodiments.

B. Glossary of Terms

Unless the context clearly requires otherwise, throughout thedescription and the

-   -   “comprise”, “comprising”, and the like are to be construed in an        inclusive sense, as opposed to an exclusive or exhaustive sense;        that is to say, in the sense of “including, but not limited to”;    -   “connected”, “coupled”, or any variant thereof, means any        connection or coupling, either direct or indirect, between two        or more elements; the coupling or connection between the        elements can be physical, logical, or a combination thereof;    -   “herein”, “above”, “below”, and words of similar import, when        used to describe this specification, shall refer to this        specification as a whole, and not to any particular portions of        this specification;    -   “or”, in reference to a list of two or more items, covers all of        the following interpretations of the word: any of the items in        the list, all of the items in the list, and any combination of        the items in the list;    -   the use of the masculine can refer to masculine, feminine or        both;    -   where numerical values are given, they are specified to the        nearest significant figure;    -   the singular forms “a”, “an”, and “the” also include the meaning        of any appropriate plural forms.

Unless the context clearly requires otherwise, throughout thedescription and the claims:

Words that indicate directions such as “vertical”, “transverse”,“horizontal”, “upward”, “downward”, “forward”, “backward”, “inward”,“outward”, “vertical”, “transverse”, “left”, “right”, “front”, “back”,“top”, “bottom”, “below”, “above”, “under”, and the like, used in thisdescription and any accompanying claims (where present), depend on thespecific orientation of the apparatus described and illustrated. Thesubject matter described herein may assume various alternativeorientations. Accordingly, these directional terms are not strictlydefined and should not be interpreted narrowly.

The term “2D-mode” refers to any ultrasound imaging mode that provides atwo-dimensional cross-sectional view of body tissue, and may includeB-mode, a combined B-mode/Color Doppler mode, or a combined B-mode/PowerDoppler mode.

The term “B-mode” refers to the brightness mode of an ultrasoundscanner, which displays the acoustic impedance of a two-dimensionalcross-section of body tissue.

The term “Spectral Doppler” refers to a Doppler imaging mode of anultrasound scanner using a single focused line to sample data at a givenregion (for example, in a blood vessel to visualize blood velocity).

The term “PW” refers to a pulsed wave Doppler imaging mode, which usestime of flight calculations to obtain signals from a given region,showing direction and speed through a one-dimensional spectrum that isupdated over time.

The term “CW” refers to a continuous wave Doppler mode, whichcontinuously transmits and receives at a single region to obtainsignals, and can be used for high speed blood flow measurements.

The term “Color” or “Color Doppler” refers to a color Doppler imagingmode that characterizes blood flow across a 2-dimensional image, showingdirection and speed.

The term “Power” or “Power Doppler” refers to a power Doppler imagingmode that characterizes blood flow across a 2-dimensional image, showingintensity but not direction or speed.

The term “AI model” means a mathematical or statistical model that maybe generated through artificial intelligence techniques such as machinelearning and/or deep learning. For example, these techniques may involveinputting labeled or classified data into a neural network algorithm fortraining, so as to generate a model that can make predictions ordecisions on new data without being explicitly programmed to do so.Different software tools (e.g., TensorFlow™, PyTorch™, Keras™) may beused to perform machine learning processes.

The term “module” can refer to any component in this invention and toany or all of the features of the invention without limitation. A modulemay be a software, firmware or hardware module, and may be located, forexample, in the ultrasound scanner, a display device or a server.

The term “communications network” can include both a mobile network anddata network without limiting the term's meaning, and includes the useof wireless (e.g. 2G, 3G, 4G, 5G, WiFi™, WiMAX™, Wireless USB (UniversalSerial Bus), Zigbee™, Bluetooth™ and satellite), and/or hard wiredconnections such as local, internet, ADSL (Asymmetrical DigitalSubscriber Line), DSL (Digital Subscriber Line), cable modem, T1, T3,fiber-optic, dial-up modem, television cable, and may includeconnections to flash memory data cards and/or USB memory sticks whereappropriate. A communications network could also mean dedicatedconnections between computing devices and electronic components, such asbuses for intra-chip communications.

The term “operator” (or “user”) may refer to the person that isoperating an ultrasound scanner (e.g., a clinician, medical personnel, asonographer, ultrasound student, ultrasonographer and/or ultrasoundtechnician).

The term “processor” can refer to any electronic circuit or group ofcircuits that perform calculations, and may include, for example, singleor multicore processors, multiple processors, an ASIC (ApplicationSpecific Integrated Circuit), and dedicated circuits implemented, forexample, on a reconfigurable device such as an FPGA (Field ProgrammableGate Array). A processor may perform the steps in the flowcharts andsequence diagrams, whether they are explicitly described as beingexecuted by the processor or whether the execution thereby is implicitdue to the steps being described as performed by the system, a device,code or a module. The processor, if comprised of multiple processors,may be located together or geographically separate from each other. Theterm includes virtual processors and machine instances as in cloudcomputing or local virtualization, which are ultimately grounded inphysical processors.

The term “scan convert”, “scan conversion”, or any of its grammaticalforms refers to the construction of an ultrasound media, such as a stillimage or a video, from lines of ultrasound scan data representing echoesof ultrasound signals. Scan conversion may involve converting beamsand/or vectors of acoustic scan data which are in polar (R-theta)coordinates to cartesian (X-Y) coordinates.

The term “system” when used herein, and not otherwise qualified, refersto an ultrasound imaging system, the system being a subject of thepresent invention. In various embodiments, the system may include anultrasound machine (including a display and one or more transducers); anultrasound scanner and a display device; and/or an ultrasound scanner,display device and a server.

The term “ultrasound image frame” (or “image frame” or “ultrasoundframe”) refers to a frame of post-scan conversion data that is suitablefor rendering an ultrasound image on a screen or other display device.

Embodiments of the invention may be implemented using specificallydesigned hardware, configurable hardware, programmable data processorsconfigured by the provision of software (which may optionally comprise“firmware”) capable of executing on the data processors, special purposecomputers or data processors that are specifically programmed,configured, or constructed to perform one or more steps in a method asexplained in detail herein and/or combinations of two or more of these.Examples of specifically designed hardware are: logic circuits,application-specific integrated circuits (“ASICs”), large scaleintegrated circuits (“LSIs”), very large scale integrated circuits(“VLSIs”), and the like. Examples of configurable hardware are: one ormore programmable logic devices such as programmable array logic(“PALs”), programmable logic arrays (“PLAs”), and field programmablegate arrays (“FPGAs”). Examples of programmable data processors are:microprocessors, digital signal processors (“DSPs”), embeddedprocessors, graphics processors, math co-processors, general purposecomputers, server computers, cloud computers, mainframe computers,computer workstations, and the like. For example, one or more dataprocessors in a control circuit for a device may implement methods asdescribed herein by executing software instructions in a program memoryaccessible to the processors.

For example, while processes or blocks are presented in a given orderherein, alternative examples may perform routines having steps, oremploy systems having blocks, in a different order, and some processesor blocks may be deleted, moved, added, subdivided, combined, and/ormodified to provide alternative or subcombinations. Each of theseprocesses or blocks may be implemented in a variety of different ways.Also, while processes or blocks are at times shown as being performed inseries, these processes or blocks may instead be performed in parallel,or may be performed at different times.

The invention may also be provided in the form of a program product. Theprogram product may comprise any non-transitory medium which carries aset of computer-readable instructions which, when executed by a dataprocessor (e.g., in a controller and/or ultrasound processor in anultrasound machine), cause the data processor to execute a method of theinvention. Program products according to the invention may be in any ofa wide variety of forms. The program product may comprise, for example,non-transitory media such as magnetic data storage media includingfloppy diskettes, hard disk drives, optical data storage media includingCD ROMs, DVDs, electronic data storage media including ROMs, flash RAM,EPROMs, hardwired or preprogrammed chips (e.g., EEPROM semiconductorchips), nanotechnology memory, or the like. The computer-readablesignals on the program product may optionally be compressed orencrypted.

Where a component (e.g. a software module, processor, assembly, device,circuit, etc.) is referred to above, unless otherwise indicated,reference to that component (including a reference to a “means”) shouldbe interpreted as including as equivalents of that component anycomponent which performs the function of the described component (i.e.,that is functionally equivalent), including components which are notstructurally equivalent to the disclosed structure which performs thefunction in the illustrated exemplary embodiments of the invention.

Specific examples of systems, methods and apparatus have been describedherein for purposes of illustration. These are only examples. Thetechnology provided herein can be applied to systems other than theexample systems described above. Many alterations, modifications,additions, omissions, and permutations are possible within the practiceof this invention. This invention includes variations on describedembodiments that would be apparent to the skilled addressee, includingvariations obtained by: replacing features, elements and/or acts withequivalent features, elements and/or acts; mixing and matching offeatures, elements and/or acts from different embodiments; combiningfeatures, elements and/or acts from embodiments as described herein withfeatures, elements and/or acts of other technology; and/or omittingcombining features, elements and/or acts from described embodiments.

To aid the Patent Office and any readers of any patent issued on thisapplication in interpreting the claims appended hereto, applicant wishesto note that they do not intend any of the appended claims or claimelements to invoke 35 U.S.C. 112(f) unless the words “means for” or“step for” are explicitly used in the particular claim.

It is therefore intended that the following appended claims and claimshereafter introduced are interpreted to include all such modifications,permutations, additions, omissions, and sub-combinations as mayreasonably be inferred. The scope of the claims should not be limited bythe preferred embodiments set forth in the examples but should be giventhe broadest interpretation consistent with the description as a whole.

C. Claim Support

In a first broad aspect of the present disclosure, there is provided amethod for positioning a gate on an ultrasound image generated duringscanning of an anatomical feature using an ultrasound scanner, said gateat least defining an optimal location of a Doppler mode signal in atissue, the method comprising: deploying an artificial intelligence (AI)model to execute on a computing device communicably connected to theultrasound scanner, wherein the AI model is trained so that when the AImodel is deployed, the computing device generates a prediction of atleast one of an optimal position, size, or angle for the gate on theultrasound image generated during ultrasound scanning of the anatomicalfeature; acquiring, at the computing device, a new ultrasound imageduring ultrasound scanning; processing, using the AI model, the newultrasound image to generate a prediction of one or more of an optimalgate position, size and angle (the “predicted optimized gate”); andemploying the predicted optimized gate to enable corresponding Dopplermode signals.

In some embodiments, the method additionally comprises reprocessing theAI model against subsequently acquired ultrasound images atpre-determined intervals to update the predicted optimized gate. In someembodiments, the update is acceptable only within a confidencethreshold.

In some embodiments, the method additionally comprises reprocessing theAI model to against subsequently acquired ultrasound images to updatethe predicted optimized gate, such reprocessing being triggered when atleast one of the gate position, size and angle has changed beyond athreshold amount with respect to the subsequently acquired ultrasoundimages.

In some embodiments, the method comprises training the AI model usingultrasound images generated in one of B-mode (two-dimensional imagingmode) and Doppler mode.

In some embodiments, the method additionally comprises, a method ofupdating the gate as follows: displaying on a user interface of thecomputing device a live spectral Doppler mode (“SD-mode”) ultrasoundspectrum that corresponds to the predicted optimized gate; receivinginput to update to a new predicted optimized gate; capturing atwo-dimensional (2D) imaging mode (“2D mode”) ultrasound image(“captured image”); applying the AI model to the captured image togenerate a prediction of one or more of an optimal updated gateposition, size and angle (the “updated optimized gate”); employing theupdated optimized gate to enable corresponding SD-mode signals; anddisplaying a live-SD mode ultrasound spectrum that corresponds to theupdated optimized gate.

In some embodiments, the method additionally provides that receivinginput may be via at least one of the following modalities: a button, atouch-sensitive region of the user interface, a dial, a slider, a draggesture, a voice command, a keyboard, a mouse, a trackpad, a touchpad,or any combination thereof.

In some embodiments, the method comprises training the AI model with oneor more of the following: i) supervised learning; ii) previouslylabelled ultrasound image datasets; and iii) cloud stored data.

In some embodiments, the method comprises training the AI model with aplurality of training ultrasound frames, each of said trainingultrasound frames comprising a mask created in Doppler mode, from aplurality of manual inputs, which mask defines optimal gate parameters.

In some embodiments, the method provides that, when processing the newultrasound image using the AI model, the ultrasound imaging data isprocessed on at least one of: i) a per pixel basis, and the probabilityof optimal gate placement is generated on a per pixel basis and ii) aline sample basis, and the probability of optimal gate placement isgenerated on a line sample basis.

In some embodiments, the method provides that the anatomical feature isselected from group consisting of carotid artery, subclavian artery,axillary artery, brachial artery, radial artery, ulnar artery, aorta,hypergastic artery, external iliac artery, femoral artery, poplitealartery, anterior tibial artery, arteria dorsalis celiac artery, cysticartery, common hepatic artery (hepatic artery proper, gastric duodenalartery, right gastric artery), right gastroepiploic artery, superiorpancreaticoduodenal artery, inferior pancreaticoduodenal artery, pedisartery, posterior tibial artery, ophthalmic artery, retinal artery,heart (including fetal heart) and umbilical cord.

In a second broad aspect of the present disclosure, there is provided amethod for positioning a color box on an ultrasound image generatedduring ultrasound scanning of an anatomical feature, said color box atleast defining an optimal location of a color Doppler mode signal in atissue, the method comprising: deploying an artificial intelligence (AI)model to execute on a computing device communicably connected to theultrasound scanner, wherein the AI model is trained so that when the AImodel is deployed, the computing device generates a prediction ofoptimal color box placement for the color box, on the ultrasound image,during ultrasound scanning of the anatomical feature; acquiring, at thecomputing device, a new ultrasound image during ultrasound scanning;processing, using the AI model, the new ultrasound image to generate aprediction of the optimal new color box position; and employing the newcolor box position to enable corresponding color Doppler mode signals.

In a third broad aspect of the present disclosure, there is provided amethod for employing an AI model which is trained both: i) to identifyat least one of an optimal position, size, and angle of trained gates inultrasound imaging data, such that when deployed, the computing devicegenerates a prediction of at least one of an optimal position, size, andangle for a new gate on a new ultrasound image, during ultrasoundscanning of an anatomical vascular feature and ii) to identify, withregard to tissue, an optimal placement of the color box on ultrasoundimaging data such that when deployed, the computing device generates aprediction of optimal color box placement for a new color box, on a newultrasound image, during ultrasound scanning of the anatomical feature,such AI model (the “combined AI model”) predicting both optimal gatecharacteristics and color box location to employ corresponding Dopplermode signals.

In a fourth broad aspect of the present disclosure, there is provided anultrasound system for automatically positioning a gate on an ultrasoundimage, during ultrasound scanning of an anatomical feature using anultrasound scanner, said gate at least defining an optimal location of aDoppler mode signal in a tissue, said ultrasound system comprising: anultrasound scanner configured to acquire a plurality of new ultrasoundframes; a processor that is communicatively connected to the ultrasoundscanner and configured to: process each new ultrasound frame of aplurality of new ultrasound frames against an artificial intelligence(“AI”) model, wherein said AI model is trained so that when the AI modelis deployed, it identifies at least one of an optimal position, size,and angle of a trained gate in the ultrasound imaging data; acquire thenew ultrasound image during ultrasound scanning; process, using the AImodel, the new ultrasound image to generate a prediction of one or moreof the optimal new gate position, size and angle (the “predictedoptimized gate”); employ the predicted optimized gate to enablecorresponding Doppler mode signals; and a display device configured todisplay one or more of the ultrasound frames and the Doppler modesignals to a system user.

In some embodiments, in the ultrasound system, the display devicecomprises a user interface comprising: i) an input module that iscommunicatively connected to the ultrasound scanner, while theultrasound scanner is operating in SD-mode; ii) a live SD-modeultrasound spectrum that corresponds to the predicted optimized gate;said input module providing direction to the processor to update to anew predicted optimized gate such that user interface additionallydisplays iii) a captured a two-dimensional (2D) ultrasound image(“captured image”) to which is applied a prediction of an optimalupdated gate position, size and angle (the “updated optimized gate”);and iv) a live-SD mode ultrasound spectrum that corresponds to theupdated optimized gate.

In some embodiments, in the ultrasound system, the AI model is trainedwith a plurality of training ultrasound frames, each of said trainingultrasound frames comprising a mask created in Doppler mode, from aplurality of manual inputs, which mask defines optimal gate parameters.

In a fifth broad aspect of the present disclosure, there is provided acomputer-readable media storing computer-readable instructions, forautomatically positioning a gate on an ultrasound image, duringultrasound scanning of an anatomical feature using an ultrasoundscanner, said gate at least defining an optimal location of a Dopplermode signal in a tissue, said computer-readable media storingcomputer-readable instructions, when executed by a processor cause theprocessor to: process each ultrasound frame of a plurality of ultrasoundframes against an artificial intelligence (“AI”) model, wherein said AImodel is trained so that when it is deployed, it identifies at least oneof an optimal position, size, and angle of a trained gate in ultrasoundimaging data; acquire a new ultrasound image during ultrasound scanning;process, using the AI model, the new ultrasound image to generate aprediction of one or more of the optimal new gate position, size andangle (the “predicted optimized gate”); and employ the predictedoptimized gate to enable corresponding Doppler mode signals.

In some embodiments, in the computer-readable media storingcomputer-readable instructions, the AI model is trained with a pluralityof training ultrasound frames, each of said training ultrasound framescomprising a mask created in Doppler mode, from a plurality of manualinputs, which mask defines optimal gate parameters.

In a sixth broad aspect of the present disclosure, there is provided aportable computing device for updating a gate on an ultrasound scannercomprising: a user interface comprising i) an input module that iscommunicatively connected to the ultrasound scanner, while theultrasound scanner is operating in SD-mode; ii) a live SD-modeultrasound spectrum that corresponds to a previously predicted optimizedgate; said input module providing direction to a processor to update toa new predicted optimized gate such that the user interface additionallydisplays iii) a captured a two-dimensional (2D) ultrasound image(“captured image”) to which is applied a prediction of an optimalupdated gate position, size and angle (the “updated optimized gate”);and iv) a live-SD mode ultrasound spectrum that corresponds to theupdated optimized gate.

In a seventh broad aspect of the present disclosure, there is provided acomputer-readable media storing computer-readable instructions, forautomatically positioning a color box on a new ultrasound image, duringB-ultrasound scanning of an anatomical feature using an ultrasoundscanner, said color box at least defining an optimal location of aDoppler mode signal in a tissue, said computer-readable media storingcomputer-readable instructions, when executed by a processor cause theprocessor to: process each ultrasound frame of a plurality of ultrasoundframes against an artificial intelligence (“AI”) model, wherein said AImodel is trained so that when the AI model is deployed, it identifies anoptimal color box placement in ultrasound imaging data; acquire a newultrasound image during ultrasound scanning; process, using the AImodel, the new ultrasound image to generate a prediction of an optimalcolor box placement for the color box; employ the new color box toenable corresponding Doppler mode signals.

What is claimed is:
 1. A method for positioning a gate on an ultrasoundimage generated during scanning of an anatomical feature using anultrasound scanner, said gate at least defining an optimal location of aDoppler mode signal in a tissue, the method comprising: deploying anartificial intelligence (AI) model to execute on a computing devicecommunicably connected to the ultrasound scanner, wherein the AI modelis trained so that when the AI model is deployed, the computing devicegenerates a prediction of at least one of an optimal position, size, orangle for the gate on the ultrasound image generated during ultrasoundscanning of the anatomical feature; acquiring, at the computing device,a new ultrasound image during ultrasound scanning; processing, using theAI model, the new ultrasound image to generate a prediction of one ormore of an optimal gate position, size and angle (the “predictedoptimized gate”); and employing the predicted optimized gate to enablecorresponding Doppler mode signals.
 2. The method of claim 1additionally comprising reprocessing the AI model against subsequentlyacquired ultrasound images at pre-determined intervals to update thepredicted optimized gate.
 3. The method of claim 2, wherein the updateis acceptable only within a confidence threshold.
 4. The method of claim1 comprising reprocessing the AI model to against subsequently acquiredultrasound images to update the predicted optimized gate, suchreprocessing being triggered when at least one of the gate position,size and angle has changed beyond a threshold amount with respect to thesubsequently acquired ultrasound images.
 5. The method of claim 1comprising training the AI model using ultrasound images generated inone of B-mode (two-dimensional imaging mode) and Doppler mode.
 6. Themethod of claim 1 additionally comprising a method of updating the gateas follows: displaying on a user interface of the computing device alive spectral Doppler mode (“SD-mode”) ultrasound spectrum thatcorresponds to the predicted optimized gate; receiving input to updateto a new predicted optimized gate; capturing a two-dimensional (2D)imaging mode (“2D mode”) ultrasound image (“captured image”); applyingthe AI model to the captured image to generate a prediction of one ormore of an optimal updated gate position, size and angle (the “updatedoptimized gate”); employing the updated optimized gate to enablecorresponding SD-mode signals; and displaying a live-SD mode ultrasoundspectrum that corresponds to the updated optimized gate.
 7. The methodof claim 6 wherein the input is user directed and is via at least one ofthe following modalities: a button, a touch-sensitive region of the userinterface, a dial, a slider, a drag gesture, a voice command, akeyboard, a mouse, a trackpad, a touchpad, or any combination thereof.8. The method of claim 6 wherein the input is not user directed and isgenerated in response to a deficiency in SD-mode ultrasound signals. 9.The method of claim 6 wherein the user interface additionally displays afrozen 2D mode ultrasound image.
 10. The method of claim 1 comprisingtraining the AI model with one or more of the following: i) supervisedlearning; ii) previously labelled ultrasound image datasets; and iii)cloud stored data.
 11. The method of claim 1 comprising training the AImodel with a plurality of training ultrasound frames, each of saidtraining ultrasound frames comprising a mask created in Doppler mode,from a plurality of manual inputs, which mask defines optimal gateparameters.
 12. The method of claim 1 wherein when processing the newultrasound image using the AI model, the ultrasound imaging data isprocessed on at least one of: i) a per pixel basis, and the probabilityof optimal gate placement is generated on a per pixel basis and ii) aline sample basis, and the probability of optimal gate placement isgenerated on a line sample basis.
 13. The method of claim 1 wherein theanatomical feature is selected from group consisting of carotid artery,subclavian artery, axillary artery, brachial artery, radial artery,ulnar artery, aorta, hypergastic artery, external iliac artery, femoralartery, popliteal artery, anterior tibial artery, arteria dorsalisceliac artery, cystic artery, common hepatic artery (hepatic arteryproper, gastric duodenal artery, right gastric artery), rightgastroepiploic artery, superior pancreaticoduodenal artery, inferiorpancreaticoduodenal artery, pedis artery, posterior tibial artery,ophthalmic artery, retinal artery, heart (including fetal heart) andumbilical cord.
 14. A method for positioning a color box on anultrasound image generated during ultrasound scanning of an anatomicalfeature, said color box at least defining an optimal location of a colorDoppler mode signal in a tissue, the method comprising: deploying anartificial intelligence (AI) model to execute on a computing devicecommunicably connected to the ultrasound scanner, wherein the AI modelis trained so that when the AI model is deployed, the computing devicegenerates a prediction of optimal color box placement for the color box,on the ultrasound image, during ultrasound scanning of the anatomicalfeature; acquiring, at the computing device, a new ultrasound imageduring ultrasound scanning; processing, using the AI model, the newultrasound image to generate a prediction of the optimal new color boxposition; and employing the new color box position to enablecorresponding color Doppler mode signals.
 15. The method of claims 1 and14, comprising a single AI model which is trained both: i) to identifyat least one of an optimal position, size, and angle of trained gates invascular ultrasound imaging data, such that when deployed, the computingdevice generates a prediction of at least one of an optimal position,size, and angle for a new gate on a new ultrasound image, duringultrasound scanning of the anatomical feature and ii) to identify, withregard to tissue, an optimal placement of the color box on theultrasound imaging data such that when deployed, the computing devicegenerates a prediction of optimal color box placement for a new colorbox, on a new ultrasound image, during ultrasound scanning of theanatomical feature, such AI model (the “combined AI model”) predictingboth optimal gate characteristics and color box location to employcorresponding Doppler mode signals.
 16. An ultrasound system forautomatically positioning a gate on an ultrasound image, duringultrasound scanning of a anatomical feature using an ultrasound scanner,said gate at least defining an optimal location of a Doppler mode signalin a tissue, said ultrasound system comprising: an ultrasound scannerconfigured to acquire a plurality of new ultrasound frames; a processorthat is communicatively connected to the ultrasound scanner andconfigured to: process each new ultrasound frame of a plurality of newultrasound frames against an artificial intelligence (“AI”) model,wherein said AI model is trained so that when the AI model is deployed,it identifies at least one of an optimal position, size, and angle of atrained gate in the ultrasound imaging data; acquire the new ultrasoundimage during ultrasound scanning; process, using the AI model, the newultrasound image to generate a prediction of one or more of the optimalnew gate position, size and angle (the “predicted optimized gate”);employ the predicted optimized gate to enable corresponding Doppler modesignals; and a display device configured to display one or more of theultrasound frames and the Doppler mode signals to a system user.
 17. Theultrasound system of claim 16 wherein the display device comprises auser interface comprising: i) an input module that is communicativelyconnected to the ultrasound scanner, while the ultrasound scanner isoperating in SD-mode; ii) a live SD-mode ultrasound spectrum thatcorresponds to the predicted optimized gate; said input module providingdirection to the processor to update to a new predicted optimized gatesuch that user interface additionally displays iii) a captured atwo-dimensional (2D) ultrasound image (“captured image”) to which isapplied a prediction of an optimal updated gate position, size and angle(the “updated optimized gate”); and iv) a live-SD mode ultrasoundspectrum that corresponds to the updated optimized gate.
 18. Theultrasound system of claim 16 wherein the AI model is trained with aplurality of training ultrasound frames, each of said trainingultrasound frames comprising a mask created in Doppler mode, from aplurality of manual inputs, which mask defines optimal gate parameters.19. The ultrasound system of claim 17 wherein the input module is userdirected and a signal to update the gate is via at least one of thefollowing modalities: a button, a touch-sensitive region of the userinterface, a dial, a slider, a drag gesture, a voice command, akeyboard, a mouse, a trackpad, a touchpad, or any combination thereof.20. The ultrasound system of claim 16 wherein the input module is notuser directed and a signal to update the gate is generated in responseto a deficiency in SD-mode ultrasound signals.